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DART Automated Test Execution Technology Overview 

What is Tyrant? 

Tyrant is the name given by Lockheed Martin Advanced Technology Labs to a suite of technology it 
developed for running automated software tests. It is one of the major components of the complete 
DART (Dynamic Automated Range Testing) tool suite. (The other major component handles building out 
cyber test ranges.) The Tyrant suite allows multiple simultaneous developers and testers to run tests in a 
reproducible manner across a wide array of resources. The Tyrant suite also allows administrators to 
manage these test resources. Using Tyrant technologies, one can encode the logic of a test to perform 
against one or more test resources as well as the logic to determine whether the test is a success or 
failure. One can then run this test against a specific set of resources (i.e. the resources needed to run a 
single instance of the test), or have multiple instances of the test run against a diverse range of resources 
to evaluate the functionality of a system-under-test against the whole range of systems it may be run on 
in the real world. Finally, multiple developers can perform these tests simultaneously against a shared 
set of resources without conflicting with each other. 

Technical Components Overview 

Tyrant is made up of six central technical components: 

• Palantir (Testing Interface) 

• Undermine (Test Script Harness) + Test Scripts 

• Overmind (Test Script Scheduler) + Test Plans 

• Overview (Test Resource/Result GUI) 

• Reaper (Test Resource Sanitizer) 

• Remote Commit (Remote Job Submission) 

Each component is standalone from the others, allowing for each user to determine the combination of 
components most appropriate for their testing scenarios. 

Palantir 

Palantir provides a common, cross-platform test interface to each of the computer resources in the test 
environment that are running commodity operating systems (e.g. Windows, Linux, OSX, FreeBSD, etc.). 
This allows test script writers to write more cross-platform test scripts, expecting a consistent interface 
(e.g. “put file”, “execute”, “spawn”, etc.) on each of the computers needed for the user’s test scenario. 
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Undermine + Test Scripts 

Undermine is a test script harness for executing the user’s test scenarios. The user will encode their test 
procedure in a “test script”. Then he/she will run that test script via Undermine, which will effectively 
automate their procedure. By convention Undermine performs the actions on the test resources via 
Palantir if it exists on the test resource. 


TEST RESOURCES 



Overmind examines the set of test resources in the Test Resource section of the Overmind Database and 
schedules tests to be run across those resources. The user describes what tests they want to run, the 
resource constraints, and the desired iteration and/or replication of each test script in a “test plan”. 
Multiple users can submit test plans concurrently, and Overmind will schedule each possible test to run 
when possible. By convention Overmind runs an instance of Undermine for each test that needs to be 
performed. When tests are completed Overmind puts the test results into the Test Result section of the 
Overmind Database and marks the resource as requiring sanitization in the Test Resource section of the 
Overmind Database. 
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Overview 

Overview is the web-based GUI that allows a user to view currently running and past test results in the 
Test Result section of the Overmind Database previously inserted by Overmind as well as graphically 
manage the resources in the Test Resource section of the Overmind Database. 
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Reaper 

Reaper monitors the Test Resources section of the Overmind Database for resources that require 
sanitization and performs a custom action to sanitize the resource. The custom action is often 
something as simple as reverting a VM to a previous snapshot, but it can be as custom as rebuilding the 
entire operating system on that resource from scratch. Different sanitization modules can be specified 
for each resource if desired. By convention, the same sanitization module is used for all of the resources 
in the same hardware class. After the sanitization process completes, Reaper marks the resource as 
clean in the Test Resource section of the Overmind Database, indicating to Overmind that the resource 
can be considered for use in a test again. 
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Remote Commit (Remote Job Submission) 

While there are many ways to install and configure the Tyrant software, one very common setup is the 
one that allows for multiple users to submit and run their own tests on a set of common testing 
resources. 

In this scenario each user has their own local copy of Overmind, Undermine, his/her tool or System 
Under Test (SUT), and test scripts and test plans relative to that SUT. Each user runs the 
remote_COmmit tool to copy those five components from their local box to the main testing server, 
start Overmind, and submit the necessary test plans. This ensures that each user can be running 
completely different tests from every other user. This also allows each user to continue their 
development of any of those five components without affecting currently running tests. 

Each user can then browse to the Overview web GUI to monitor progress of their running tests. 

The following picture illustrates all of the Tyrant components working together to execute this CONOR 
(“P” == Palantir, “UM” == Undermine, “OM” == Overmind) 
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For the Developer 

For the developer, Tyrant provides the tools to create and run automated test scripts (in undermine) and 
test plans (in overmind), view test results, and work simultaneously on a range of resources. This 
manual covers how to use a range of ESXi virtual machine resources previously set up by an 
administrator and how to create and run tests. 

Repository Structure 

The core of Tyrant is provided in two Mercurial repositories: tybase and tyworkflow. The tybase 
repository provides the tools used for running single tests. The tyworkflow repository provides the tools 
used for running tests across ranges of resources and managing these ranges. The tybase repository is 
standalone, but the tyworkflow repository has dependencies on tybase. 

With the split of functionality between tybase and tyworkflow, it can sometimes be difficult to 
determine which repository you should be in to perform certain operations. Unless specifically directed 
otherwise by this manual, the following are a few general rules to help you determine the correct place 
to perform certain operations: 

• If the operation relates to running an individual test against specific test resources without use 
of any range scheduling (e.g. overmind), you should be in tybase. 

• If the operation relates to linking in collections of test scripts (leafbags), you should be in tybase. 

• If the operation relates to scheduling tests to run on a range or managing a range of shared 
resources, you should be in tyworkflow. 

The emissary repository adds the ability to perform operations on Windows-based resources as a regular 
user, rather than the SYSTEM context in which normal palantir operations run. 

The magnum repository contains code used for testing with PSPs (Personal Security Products) and for 
automatic software updating. It also contains support for detecting PSP events by watching a resource’s 
screen for changes. Related to magnum are the PIL repositories (PIL-linux-i686 and PIL-linux-x86_64). 
These provide the Python Imaging Library, which is used by magnum event detection to find changes 
between screenshots. Two exist because they contain compiled code which needs to match the 
architecture of the system it will run on. 

In-depth descriptions of the contents of each repository are included in Appendix B . 

Tools 

This manual covers the following Tyrant tools, which will be used and configured by the range 
administrator (with the repository containing each tool in parentheses): 

palantir (tybase): A component which runs on each test resource, allowing the resource to be 
controlled by test scripts via a TCP connection. Palantir serves on ports 51134 and 51135 (for 
Windows) and 51134 (for Linux). 
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undermine (tybase): The component which runs a single instance of a test. Undermine connects via 
palantir to the resource(s) used by a test and runs the given test script. 

overmind (tyworkflow): The component which schedules tests to run simultaneously. Overmind 
uses a database of test resources and schedules these resources for incoming tests. 

reaper (tyworkflow): The component which handles reverting a resource to a clean state prior to 
running a test. What this means depends upon the reaper module in use for a given resource. 
For a virtual machine, this usually means reverting to a snapshot, but for a physical computer, 
this could mean using some imaging or auto-building solution to build out a certain OS 
configuration on a resource on-demand for a test. 

overview (tyworkflow): The component which displays test results and allows some management of 
test resources via a web-based interface. 

process_plan (tyworkflow): A command-line tool used to process test plans (specifications for 
running multiple instances of tests against a specified variety of resources). This lets you submit 
plans to be run or see how many combos (specific runs of a test on specific resources and with 
specific parameters) would be run. 

remote_commit (tyworkflow): A tool which allows remote submission of tests to a central Tyrant 
environment. While overmind on its own allows simultaneous tests to be run against a range of 
resources, remote commit makes overmind useful for a team of developers running different 
sets of code. 


Directory Structure 

Because each setup will involve testing different components, users will need to create a directory 
structure similar to the following: 
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Symlinks can be made between the various repositories using the standard In - s command. 

Additionally, a user should be in the base of either tyworkflow or tybase to run specific commands. For 

example, if a user was running . /bin/ undermine, he would be doing so out of the tybase directory. 

If he was running . /bin/ remote_commit, he would be doing so out of the tyworkflow directory. 

Assumptions 

This document makes some assumptions about the type of setup desired. Specifically: 

• It is assumed that testing will be performed on VMWare ESXi virtual machines. 

• It is assumed that remote commit-style testing will be performed (with a team of developers, 
each at their own workstation, remotely submitting tests to a central server). 

• It is assumed a mysql database will be used for storing resource and test information rather than 
sqlite3. 

• The remote commit environment on the test server will run as root, and developers will submit 
their tests as root on the test server. 

• The reader is familiar with working on the Linux command line and programming in Python. 
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Environment Setup 

Before we can get to the work of writing and running tests, the environment must be set up. We will 
perform the setup now and verify its correctness in later sections. 

On your developer workstation, clone the tybase and tyworkflow repositories as siblings in the same 
directory. In the tyworkflow directory, run make. This links tyworkflow to tybase and unpacks our built- 
in python distribution. 

A few configuration files need to be modified to enable you to use the range that’s been set up for you. 

In tyworkflow, copy the file rc/def aults/db . rc to rc/. In the copied file, make the following 
changes (you may need to talk to the administrator to get this information): 

• set resource_manager/dbname to the name of the overmind database 

• set resource_manager/engine to mysql instead of sqlite3 

• set mysql/host to the hostname or IP address of the test server, which is where the overmind 
database resides 

• set mysql/user and mysql/passwd to the username and password used to access the 
database you set in resource_manager/dbname 

Also, copy rc/ defaults/ remote_commit . rc to rc/ and make the following changes: 

• remote_commit/remote_user: the user to connect to the testing server as (via SSH when 
syncing up files or running commands; defaults to root, which is typical) 

• remote_commit/remote_host: the hostname or IP address of your testing server 

• remote_commit/remote_commit_dir: the path on your testing server to the commits 
directory (/proj /testing/commits is the example used in the administrator manual). 

For the above settings, make sure you uncomment any options which you set which are currently 
commented. That is, remove the semicolon from the beginning of any option line that you modify. 

Leafnodes (Test Scripts) 

Test scripts (known as “leafnodes” in Tyrant lingo) are the main units of work performed by undermine. 
Leafnodes typically involve one or more assets with palantir installed on them (but not always). 
Leafnodes can be as simple as a Python function that operates on some input parameters, to as complex 
as a class that choreographs a set of actions on multiple remote hosts via palantir. Where practical, 
leafnodes should be made smaller rather than larger to facilitate reuse. 

Leaf node Concepts 

Leafnodes come in different types (detailed below), but share some or all of the following concepts, 
which will be helpful to keep in mind for the rest of this manual: 
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• hosts: The test hosts, or resources against which the leafnode works. Some leafnodes do not use 
any resources, but most do, as running tests on computers is a primary purpose of leafnodes. 

• inputs: If you think of the leafnode as a function, these are the arguments. 

• progress messages: Asynchronous messages the leafnode can output during the course of its 
execution. 

• result code: An indicator of the status of the leafnode returned when it's finished (e.g. success, 
failure, error). 

• result (or output): A return value from the leafnode (different from its status). 

Creating and Running a Simple Leafnode 

We’ll start by creating a simple leafnode, then having done that, delve into the details that will let you 
create more powerful test scripts. 

To start, inside yourtybase directory, create the directory path leaf bags/tutorial/tutorial 

(the duplicate tutorial in the path is intentional). In this directory, create an empty init . py 

file to make it a valid python package. 

Open up your text editor of choice and enter the following text: 

from tybase. undermine. leaf import Leaf 
from tybase. undermine. meta import leafi 

§leaf i . Def ineActuator ( ) 
class Hello(Leaf ) : 
def run(self ) : 

host = self . hosts[0] 

ret = host. execcmd( 'echo' , self .args[0] ) 
host . f write ( ' /tmp/test . txt ' , self . args[0] ) 
dat = host . fread( ' /tmp/test . txt ' ) 
if dat.strip() == ret.strip(): 

return self . SUCCESS, 'cmd output matched file contents' 
else: 

return self . FAILURE, 'cmd output differs from file contents' 

Save this as leaf bags/tutorial/tutorial/hello_world . py. 

In your web browser, navigate to the overview web interface running on the test server (e.g. 
http://testserver.example.com/overview). This is the interface by which you will later be able to view 
your test results, and will be covered in detail later. For now, click the Management link. Select a Linux 
resource whose “status” column says “avail”, check the box next to it, put your name in the “Reserve 
Name” field on the left-hand side of the page, and click “Reserve”. This reserves the machine for your 
use so that it won’t be scheduled for others’ automated tests, preventing your work from messing up 
others’ or vice versa. Note the IP address of this resource. 
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In your tybase clone, run the following command, where IP_ADDR is the IP address of the resource you 
reserved: 

bin/undermine tutorial . hello_world . Hello IP_ADDR - "hello, world" 

Here's an example of the output you should see: 

2013-09-23_16 : 51 : 02 . 44 (00191) [INF] script 22331: OUtput_dir: 

. /output/undermine/nlsheppa/2013_09_23-16_51_02_348754 

2013-09-23_16 : 51 : 02 . 44 (00191) [INF] script 22331: COMPLETION: 
success 'hello, world' 

If you look on the actual Linux VM you reserved, you'll find a file / tmp/hello . txt with contents 
“hello, world”. 

You have written and run your first leafnode. Unreserve the resource you reserved previously by going 
back to the Management page, checking the box on the resource you reserved and clicking the “Use 
Testing” button. 

Leaf nodes in Depth 

Writing Leafnodes 

The general idea of writing a leafnode is that you write some sort of callable (see below) which receives 
zero or more palantir client objects, arguments and keyword arguments, performs some operations with 
them, and then returns a result code (see below) and result value, optionally with some progress 
message along the way. The callable is decorated with various decorators to define "metadata" for the 
leafnode. 

In this section, we first illustrate the different ways of writing leafnodes, then dive in to the details of 
how to define metadata on them. 

Types of Leafnodes 

Here we cover the two main styles of leafnodes, class-based and function-based. 

Type 1: class-based leafnodes 

Class-based leafnodes are the most powerful and consist of a class which inherits from the Leaf class 
provided in tybase (by importing tybase. undermine.leaf.Leaf) and overrides certain methods (all of 
which take no arguments other than the self reference to the instance they're bound to). The methods 
which the leafnode may override are: 

• runSetup: Run before the body of the leafnode. If this raises an exception, the leafnode will stop 
and the SKIPPED result code will be returned. 

• run: The body of the leafnode. If this raises an exception, the leafnode will stop and the ERROR 
result code will be returned. Otherwise, this method must return a tuple of the result code and 
output value of the leafnode. 

• runCleanup: Run after the body of the leafnode in all cases except when the leafnode times out, 
regardless of the leafnode's result code. If runSetup has an error other than a timeout, 
runCleanup is still run (so runCleanup is similar to the finally block of try ... except ... finally). The 
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success or failure of runCleanup does not affect the final result code of the leafnode. If the 
leafnode body returns SUCCESS, the final result code will be SUCCESS even if runCleanup throws 
an exception. 

• stopHandler: Run in the case of a timeout, when the leafnode is being stopped. This method has 
a limited time (currently five minutes) in which to run, which is why it's separate from 
runCleanup, which doesn't have as small a time limit. 

In this style of leafnode, self is your reference to the currently running script. You access your host(s) 
through self.hosts, which is a list of palantir client objects (even if the leafnode only takes one host). Your 
arguments are provided initially in self.args (positionally-specified arguments) and self.kwargs 
(arguments specified with keywords) without regard to what's defined in the input parameters. You'll 
probably want to normalize your args by calling either self.normalize_args or self.normalize_kwargs in 
your run method (see below). 

Here's an example python script illustrating this type of leafnode (the DefineActuator and other 
metadata decorators will be covered later): 

from tybase . undermine . leaf import Leaf 
from tybase . undermine . meta import leafi 

@leafi . DefineActuator ( ) 
class MyLeafnode(Leaf ) : 
def runSetup(self ) : 

#here you do setup tasks, like perhaps installing some 

#supporting piece of software on an asset 

pass 

def runCleanup(self ) : 

#here you do cleanup, like perhaps deleting some temporary 

#files 

pass 

def run(self ) : 

#the body of your leafnode 

#normalize to a diet of kwargs so we get our default values 

#and everything easily accessible by name, even if the args 

#w ere given positionally 

self.kwargs = self . normalize_kwargs( ) 

#a common thing to do if you only have one host 
host = self .hosts[0] 

#do some testing stuff 

#perhaps we want to measure how much data was exchanged over 
#the network in this case, our result value is an integer; 
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#data type specification will be explained further on 
traffic_size = some_measurement_f unction ( ) 

return (self .SUCCESS, traffic_size) 

Type 2: function-based leafnodes 

When you have a simpler testing task, you might choose this second type, in which you simply write a 
python function, which you optionally decorate with some metadata. This type of leafnode is simpler, 
but also less powerful. Differences compared to class-based leafnodes are: 

• Your reference to the currently running script is stored at the context attribute of a host 
object. This unfortunately means that if your function-based leafnode takes no hosts, you will 
not have access to a reference to the currently running script and will not be able to do things 
which require it, like running a sub leafnode. 

• You cannot define setup and cleanup logic like you can with runSetup and runCleanup for class- 
based leafnodes. 

• You have less flexibility in setting the result code of your leafnode, as follows: 

o If your function raises an exception, ERROR will be returned, with the exception as the 
result value. 

0 If your function returns at all (whether True, False, a number, None, anything), then 
SUCCESS will be used. 

• Input parameters are typically defined implicitly by the function prototype, rather than explicitly 
with metadata on the leafnode. 

• Arguments are handled differently. The arguments that are passed to your function-based 
leafnode consist of the host objects, followed by the args, followed by the kwargs. Thus, if your 
leafnode is called with too many or two few hosts, you can end up with an argument you 
expected to be a host containing an arg value (too few hosts), or an arg containing a host rather 
than the arg value you expected (too many hosts). 

• Hosts, args and kwargs are accessed by the names you give them in the function prototype, as 
with any function. 

Here's an example python script illustrating a function-based leafnode: 

import tybase. undermine. meta.leafi as leafi 

def my_leafnode(hostl, host2, argl, arg2, kwargl=0, kwarg2=True) : 

#do some testing stuff 

#no matter what you return here, the result will be SUCCESS (as 
#long as you actually return and don't throw an exception) 
return True 

Leafnode Metadata 

Leafnode metadata is how you define things like what kinds of assets your leafnode works against, what 
kinds of data it takes on input and output, whether it provides asset properties, etc. Metadata is set on a 
leafnode by decorating the leafnode with decorators provided in the tybase.undermine.meta. leafi 
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module. Metadata is not strictly required to run a leafnode, but it is advised, and it is necessary to take 
advantage of certain advanced leafnode features. This section's subsections explain the provided 
metadata decorators organized by the purpose they serve, with the name of the decorator in 
parentheses. 

Defining the Leafnode Purpose (DefineActuator, DefineSensor, DefineProcessor) 

These three mutually exclusive decorators describe the function the leafnode serves and how it will 
interact (or not) with any assets it uses. Currently the usage of these decorators is only by convention; 
they don't do anything special to the leafnode you put them on (except for DefineSensor with asset 
properties). However, leafnodes need to have some type of metadata, and putting one of these on the 
leafnode is a good way to satisfy that requirement. 

The convention for these decorators is: 

• DefineActuator: leafnodes that make changes to an asset (e.g. delete a file, install a piece of 
software, etc) 

• DefineSensor: leafnodes that only query information on an asset, not make changes to it. If you 
want your leafnode to assert (provide) asset properties, it must have this decorator. 

• DefineProcessor: leafnodes that do not care about what assets they receive, and may not even 
take any hosts at all 

Defining Input Parameters (Inputs) 

This decorator takes as its arguments tuples defining 

• the name of an input parameter 

• its data type (see below for valid types and how to specify them) 

• optionally a default value for the parameter. 

This decorator only makes sense for the first type of leafnode (class-based), since the other two types 
depend on the arguments defined in the function prototype. 

An example usage is: 

import tybase. undermine. meta.leafi as leafi 

@leafi. Inputs( 

( 'num_runs' , int), 

('interval', float, 5.0), 

('path', str, 'C:\\test_dir ' ), 

( ' quick_run ' , bool, False) 

) 

class MyLeafnode . . . 

Input and Output Data Types 

Leaf node parameter data types can be any scalar type that can be pickled, as well as lists or structs. 
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For scalar types, the data type definition (the second field of the input parameter tuple, or the argument 
to the FinalOutput decorator) is simply the type. For example: 

@leafi . Inputs ( ( ' num_runs ' , int, 5) ) (defines a single input parameter of type int 
named num_runs with default value 5) 

@leafi. FinalOutput (bool) (defines a result value of type bool) 

Typical scalar types are str, int, float and bool. 

For complex types, you show the data types of the scalar parts of the complex types in the context of 
that type (as a list for lists, as a diet for structs). Also, complex types may be nested. For example: 

@leafi . Inputs ( ( ' animals ' , [str] ) ) (defines a single input parameter which will be 
a list of strings named animals and have no default value) 

§leaf i. FinalOutput ({' MemTotal ' : int, 'MemFree': int, 'Swapfile' : str}) 
(defines a result value which will be a struct with three fields of type int, int and str, respectively) 

§leafi.FinalOutput([{'name' : str, 'lat': float, 'long': float}] ) (defines a 
result value which will be a list of structs, each representing a city) 

@leaf i. FinalOutput ({' size ' : int, 'files': [str]}) (defines a result value 
which will be a struct of a size value and a list of filenames [perhaps this is the size and contents 
of some archive file the leafnode processed]) 

Flowever, valid leaf node lists and structs are more limited than what can be expressed in Python lists 
and diets, as follows: 

Lists 

Lists are defined by giving the type of the scalar values of the list in list context in the input/output 
definition, as seen in the above table. Thus, every element contained in a list must be of the same type. 

If you need to pass a static set of values of different types, consider using a struct instead. If you really 
need to pass a variable number of items of different types, consider (1) a list of structs, or (2) multiple 
lists, each of which contains a different type. 

Structs 

Structs are defined by giving a diet whose keys are the names of the struct fields and whose values are 
the scalar types for each field. The difference between the leafnode struct type and regular Python diets 
is that structs are defined with a static set of fields. If you really need to input (or output) a dict-like data 
structure, here are a couple of options: 

('keyValsl', [[str]]), # only if key and value types are the same 
('keyVals2', [{'key':str, ’val':int}]) # different types for key & value 
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Even with the above alternatives, leafnodes are still restricted in that they cannot input or output 
arbitrary types. 

Deriving Inputs from Function Introspection (Derivelnputs) 

For function- and method-based leafnodes, where input parameters are based on the function/method 
definition, this decorator will use function introspection to automatically generate input parameter 
metadata (as you would specify with the Inputs decorator for class-based leafnodes) from the 
function/method definition. You simply provide this decorator with no arguments, like so: 

@leafi.Derivelnputs( ) 
def my_leafnode( . . . ) 

Defining the Result Data Type (FinalOutput) 

The data type of the result value, aka output, is defined using the FinalOutput decorator, which takes the 
data type specification as its argument. This decorator uses the same specification as the Input 
decorator, as explained above. 

Defining the Progress Message Data Type (ProgressOutput) 

The progress message data type is defined just as with the result data type, but using the 
ProgressOutput decorator instead of the FinalOutput decorator. 

Defining Leafnode Alias (Alias inside of a Define*) 

The Define* decorators do have another purpose. Inside of a Define* decorator, you can also specify an 
alias with the Alias class provided in the leafi module. This is used with polymorphic leafnodes. You 
define an alias like so: 

@leafi . DefineActuator (leafi . Alias( ' uber_leaf node ' ) ) 
def my_leafnode( . . . ) 

Defining the Leafnode "Subjects" (assets to run against) (Subject) 

Using the Subject decorator, you can define constraints to restrict what your leafnode can run on. These 
constraints utilize asset properties (a list of which can be found here). In the Subject decorator, with the 
constraints keyword, you specify a list of constraint comparisons with the Prop, AndProp and OrProp 
classes provided in the leafi module. All the elements of the constraints list must match for the leafnode 
to be allowed to run. 

Some examples: 

Require 64-bit Windows 7: 

@leafi.Subject( 

constraints=( 

leafi . Prop( ' sw. os . architecture ' ) =- 'x86_64', 
leafi . Prop( ' sw. os . name ' ) == '6.1', 
leafi. Prop( 'sw. os. family' ) == 'Windows' 
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) 

) 

Require 64-bit Windows 7 (illustrating the use of AndProp, which is unnecessary, but valid): 

@leafi.Subject( 

constraints=( 

leafi. AndProp( 

leafi.Prop( 'sw. os. architecture' ) == 'x86_64', 
leafi. Prop( 'sw. os. name' ) ss '6.1', 
leaf i. Prop ( 'sw. os. family' ) == 'Windows' 

), 

) 

) 

Require 32-bit Windows 7 or XP (illustrating the use of OrProp): 

@leafi.Subject( 

constraints=( 

leafi . OrProp( 

leafi. Prop( 'sw. os. name' ) == '5.1', 
leafi. Prop( 'sw. os. name' ) == '6.1' 

), 

leafi . Prop( ' sw. os . architecture ' ) == 'x86_64', 

) 

) 

Defining the Default Leafnode in a Module (MainLeaf) 

If you so choose, you can mark a leafnode in a module as the default leafnode for that module. Then, 
when you specify the leafnode to run (e.g. on the undermine command line), you need only specify as 
far as the module, and undermine will automatically run the default leafnode you marked. To do this, 
place the MainLeaf decorator on the desired leafnode. 

For example, consider the example leafnode we created and ran in Creating and Running a Simple 
Leafnode . If we added the MainLeaf decorator to the Hello class, then rather than referencing the 
leafnode as tutorial . hello_world . Hello like before, you could reference it as simply 
tutorial . hello_world. 

The MainLeaf decorator would be added to the example leafnode like so: 

@leafi . DefineActuator ( ) 

@leafi. MainLeaf () 
class Hello(Leaf ) : 
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Inheriting Metadata from Parent Classes (InheritMeta) 

With the class-based style, where class inheritance is involved, this decorator can be used to inherit 
metadata defined on a parent class to the child class. You put this decorator on the class and the class 
would inherit metadata from parent classes. 

Inside the Leafnode 

At this point, we know how to structure a leafnode, but what goes in the body? 

Technically, pretty much whatever you want within the limits of python. Typically, you interact with 
palantir client objects to put or get files from assets, run commands on them, do operations influenced 
by the input parameters, and so forth. To see a list of the operations available with palantir, run 
bin/ palantir_admin -h in your tybase clone. Each of the methods documented therein are 
accessible on the host objects. For example, to put a file, you’d do: 
self . hosts[0] . put(src, dest) 

During the leafnode, you may choose to emit progress messages and at the end you return a result code 
and a result value (depending upon leafnode style). 

Emitting Progress Messages 

To emit a progress message, you call the emit Progress method on the Leaf class. If you're in a class- 
based leafnode, that means calling self . emitProgress. If you're in a function- or method-based 
leafnode, you need a reference to the currently running script, which is available on your palantir client 
objects as the context attribute (e.g. host . context . emitProgress). This method is defined as 
follows: 

def emitProgress(self , data, seq=-l, tstamp=None, spath=None, dpath=None) 
where the parameters are: 

• data: The value of the progress message, which must match the type you defined in the 
ProgressOutput decorator. 

• seq: Sequence number of the progress message (defaults to one greater than the last one, 
starting with zero) 

• t stamp: Time stamp of the progress message, defaults to the current time. 

• spath: Perhaps this means source path, but it appears to have no meaning and is rarely used, if 
at all. 

• dpath: Override the path to the file where the progress message is stored, defaults to a tile in 
the output directory whose name includes the sequence number. This is rarely used. 

Usually, you should only provide data. An example call would look like: 

self . emitProgress ( { ' currentSpeedMPH ' : 88 . 8} ) 

Returning a Result 

When in a class-based leafnode, you return a tuple of the result code and the result value. The result 
codes are constants defined on the Leaf class, as enumerated below. The result value is whatever value 
you want, of the type you defined in your FinalOutput decorator. 
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When in a function-based leafnode, as explained above, you simply return the result value and the result 
code is determined for you. 

Leafnode Result Codes 

The valid leafnode result codes are defined as attributes on the Leaf class and are as follows: 

• SUCCESS: test completed without error and returned desired results (e.g. your software works) 

• FAILURE: test completed without error and returned incorrect results (e.g. your software doesn't 
work, but your test is written properly) 

• ATTENTION: test completed without error and returned generally desired results, but something 
is fishy and you want a human to follow up 

• SKIPPED: test had an error in setup (either in basic undermine stuff like reaping an asset or in 
your optionally provided setup code) 

• ERROR: test had an error while running the body of the test (e.g. your test has a problem and 
threw an exception) 

Based on the convention of these result codes, your code should generally only explicitly return 
SUCCESS, FAILURE, or ATTENTION. If you want ERROR, you should throw an exception describing the 
error. Your leafnode body should not return SKIPPED since that is for setup problems. 

Normalizing Arguments 

Note: This only applies to class-based leafnodes. 

By default, the code that calls your leafnode's functions does nothing to set up the arguments for you. 
That is, when an instance of your leafnode is created, you get some positional arguments and some 
keyword arguments based on how your leafnode was called. Another effect of this is that default values 
are not handled at all, which means you end up with None for the value of any arguments which are not 
provided in the call to your leafnode. So that you don't have to write your own argument handling code, 
two methods (normalize_args and normalize_kwargs) have been provided. These are 
methods on the Leaf class, so you can access them simply by calling self . normalize_args or 
self . normalize_kwargs from your class-based leafnode. Both take no arguments. 

normalize_args returns the arguments to your leafnode as a list of positional arguments. 
normalize_kwargs returns the arguments as a diet keyed by input parameter name. Both set 
default values for arguments which are not provided (if you gave default values in your leafnode's 
metadata) or otherwise throw exceptions (if you did not give default values). Also, you may only call one 
of these functions (if you call both, then you may get errors about arguments being provided multiple 
times). The idea is for you to set self . args or self . kwargs to the return of the respective 
normalize function (e.g. self . args = self . normalize_args( )). If your leafnode overrides the 

init method, that would be a good place to put it, otherwise you could just put it near the 

beginning of your run method. 
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Logging Information 

Logging information may be output from a leafnode using the log attributes present on the script class 
and each host object. These log attributes are instances of the python logging module’s Logger object 
(see http://docs.python.0rg/2/library/logging.html for full details on using python logging). Logging 
entries output using a host object’s log attribute are tagged with the IP address or hostname of that 
host. All these log entries will show up in the script.log file in the leafnode instance’s output directory 
(explained further in the "Running Leafnodes” section). 

Examples: 

• Logging an informational message to the script class’s logger in a class-based leafnode: 

self . log . info("something happened") 

• Logging an informational message to the script class’s logger in a function-based leafnode 
(assuming the leafnode takes at least one host and the first parameter is named "host”): 
host . context . log . inf o( "something happened" ) 

• Logging a warning related to a specific host in a class-based leafnode: 

self . hosts[2] . log . warning("something might be wrong") 

Performing Palantir Operations as a Normal User (Windows Only) 

Normally, when you perform operations on a test resource with palantir (i.e. using methods on one of 
your host objects), that operation is run on the test resource by a palantir processing running as the 
SYSTEM user. If you have an operation that you need to run as a regular user (e.g. because you need to 
interact with the GUI on modern Windows or need the operation to run with limited user privileges), this 
can be done using functionality in the emissary repository. 

To use this, you’ll first need to link in the emissary leafbag. Clone the emissary repository into the same 
directory which contains your tybase and tyworkflow clones, then run the following commands from the 
root of tybase: 

In -s ./emissary/leaf bag leaf bags/emissary 

bin/prepare 

Then, in your test script, import the createEmissary function like so: 

from emissary. emissary import createEmissary 

Finally, in the body of your leafnode, add a line like the following: 

emhost = createEmissary(HOST_OBJ, 

domain= ' DOMAIN' , username=' USERNAME' , password= ' PASSWORD ' ) 
Replace HOST_OB J with a reference to the existing host object for the test resource on which you want 
to run operations as a regular logged-in user, and replace DOMAIN, USERNAME, and PASSWORD with 
the domain name, username, and password of the account you want to run as. If the specified user is 
not logged in, auto-login registry keys will be set and the test resource will be rebooted to cause the 
desired user to log in. If domain, username, and password are all omitted, then operations will be run as 
whatever user is currently logged in. If only domain is omitted, then the domain will be ignored when 
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determining whether the correct user is logged in and when specifying via the registry what user to 
automatically log in. 

Once this line of code has run, emhost will be a reference to an instance of palantir running as the 
specified user. This works exactly the same as any other palantir client object; it has exactly the same 
methods and properties, the only difference is what user the remote palantir server is running as. 

Storing Leafhodes (Modules and Leafbags) 

Leafnodes are stored in python modules in what's known as "leafbags". A leafbag has a very specific 
definition which must be followed: A leafbag is a directory which contains python packages. These 
python packages then contain python modules with leafnodes in them. 

When undermine runs a leafnode, the roots of all the configured leafbags are on the python path. This is 
why, when running a leafnode, you can specify it as a "python import path", like you were trying to 
import it in a python script. Deep down in the guts of undermine, that is what is actually happening. 

Structuring Leafnode Modules 

Leafnodes are stored in python modules containing one or more leafnodes and having one of several 
markers within the first 200 bytes of the file. When a module has one of these markers, we call it "leafy". 
A module must be leafy in order for it to be scanned when tybase's bin/prepare is run. This marker 
allows the leafnode scanner to quickly ignore modules that have a very low potential of containing 
leafnodes. 

Valid leafy markers are as follows: 

• the string "THIS_IS_A_LEAF_MODULE" 

• the string "#AUTOGENERATED" at the beginning of a line 

• an import involving tybase.undermine.meta.leafi, e.g. 

from tybase.undermine.meta.leafi import fooor 
import tybase . undermine . meta . leaf i 

• an import involving tybase.undermine. leaf 

• an import involving tybase.undermine. main_script 

Structuring Leafbags 

Your leafbag should have one or more levels of subdirectories and you should put leafnodes in these 
subdirectories (not in the root of the leafbag). In general, you should try to create your leafbags to either 
be entirely self-contained, or to depend on other leafbags (which you would then link in like normal, 
with no added complexity). This makes things easier in the long run. However, this is not always 
practical. If your leafbag depends on other tiles from elsewhere in your project's repository or just on 
other tiles in general, you will need to be aware of this and take it into account when using remote 
commit. See the section on leafbags with non-leafbag dependencies for how to handle this. 

Since the directories in your leafbag are being treated as python packages, they must have 
init . py tiles like any python package. When you run bin/prepare, a component of tyrant 


24 


UNCLASSIFIED 


UNCLASSIFIED 


will scan the leafbag. The leafbag scanner will only recurse into a subdirectory of the leafbag if at least 
one of the two following conditions is true: 

• the subdirectory's init . py file contains the leafbag marker (the comment #LEAFBAG) 

• the init . py file in an ancestor of the subdirectory up to but NOT including the root of 

the leafbag contains the leafbag marker with the RECURSE flag (the comment # LEAF BAG 
RECURSE). The RECURSE flag tells the scanner to go through all the subdirectories regardless of 
whether they have a leafbag marker. 

So, the simplest thing is to just put an init . py in each top-level subdirectory of your leafbag 

with the comment #LEAFBAG RECURSE. If for some reason you have directories in your leafbag 
devoid of leafnodes (such as a large third-party python module), then you might choose to not use 
recursion globally and only put the leafbag marker in specific subdirectories' init . py files. 

Leafbag structure example 

Consider an example software project (call it eproj) whose developers want to perform automated 
testing with leafnodes. This project may have a code repository (also named eproj) with a subdirectory 
called tests which is the leafbag for this project. With this in mind, consider the following partial 
directory and file structure for the example leafbag: 

• eproj / (root of the repository) 

0 tests/ (root of the leafbag) 

■ utils/ (supporting python modules, not leafnodes) 

• init . py (has no leafbag marker) 

■ net_tests/ (leafnodes for testing the software in a network) 

• init . py (contains leafbag marker) 

• test_data/ (some kind of data used for the tests and some python 
modules to work with it, but no leafnodes) 

o init . py (exists to make this a valid python package, but 

has no leafbag marker) 

■ Standalone_tests/ (leafnodes for testing the software on a single 
computer) 

• init . py (contains leafbag marker with RECURSE flag) 

• win_xp/ (leafnodes for Windows XP) 

O init .py (no leafbag marker) 

• win_7/ (leafnodes for Windows 7) 

o init .py (no leafbag marker) 

• notes/ (contains no init . py at all) 

With this structure, the scanner will 

• skip utils/ (since it has no leafbag marker, and it's a top-level subdirectory so there is no chance 
of having an ancestor with a leafbag marker with the RECURSE flag) 

• look in net_tests (since it has a leafbag marker in its init . py) 
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• skip net_tests/test_data (since it doesn't have a leafbag marker and no ancestor has the 
RECURSE flag) 

• look in standalone_tests and any subdirectories (since the init . py in standalone_tests 

has a leafbag marker with the RECURSE flag), BUT... 

• skip standalone_tests/notes (since it has no init . py at all and is therefore not a valid 

python package) 

Linking-in leafbags 

In order to use leafnodes in a leafbag with tyrant, you must link this leafbag in to your tyrant repo. The 
typical way to do this is to create a symlink in the leafbags directory of tybase that points to the leafbag 
you want to use. An alternative is to actually put your leafbag in the leafbags directory of tybase, but this 
usually doesn't make sense from an organizational standpoint because your typical project using tyrant 
has its own repository with the leafbag as a subdirectory. 

So, following the example leafbag above, assuming your current working directory is the root of tybase 
and that the eproj repo is checked out in the same directory as tyrant-dev (the two are siblings), you 
would run the following command to link in the leafbag: 

In -s . . / . . /eproj/test leafbags/eproj 

This results in a symlink called eproj in leafbags that points to the leafbag in the eproj repo. 

Mitigation of naming conflicts 

Note that leafbags can cause naming conflicts. For example, consider two projects, eproj and fproj. 
Suppose that both projects have leafbags with subdirectories named net_tests. In this case, if both 
leafbags are linked in to tyrant at the same time, a naming conflict will occur. The preferred way to 
mitigate this is to add an extra directory level in the leafbags named for the project. For example, in the 
current situation, the net_tests subdirectories that are conflicting are located at 
eproj /tests/net_tests and f pro j /test s/net_tests. To mitigate this, one could add an 
extra directory level to end up with eproj /tests/eproj/net_tests and 
fproj /tests/f pro j/net_tests, respectively. Then, leafnodes in 

eproj /tests/eproj/net_tests would be referenced with a python import path starting with 
eproj . net_tests, and those in f proj/tests/f proj/net_tests would be referenced 
starting with fproj . net_tests. 

Running Leafnodes 

This section deals with running leafnodes apart from the automated workflow and range management 
features provided by overmind. For range testing, see the sections on Test Plans and Remote Commit. 

Single Tests 

Undermine, provided in the tybase repository, is the primary tool used to run leafnodes. Undermine lets 
a user run a single instance of a test script against one or more specific resources. The basic usage of 
undermine is as follows: 

bin/undermine <leaf_spec> <host_spec> [<host_spec> ...] -- 
<args_and_kwargs> 
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The command line components are as follows: 

• leaf_spec: This is the specification of the leafnode to run, and may be given as one of the 
following: 

0 python import path to the leafnode: Since all leafbags are on the python path, you can 
give the "python import path" to your leafnode as if you were in python code trying to 
import it. For example, suppose you have a leafnode called ping in the hie 
leaf bags/my_leaf bag/utils/network_f uncs . py (relative to the root of 
tybase). Then, since leaf bags/my_leaf bag is on the python path, if you were in 
python code and wanted to import your ping leafnode, you would say import 
utils . network_f uncs . ping. Therefore, to run this leafnode, you would use 
utils . network_funcs . ping as the leaf_spec. If you put the MainLeaf 
decorator on your ping leafnode, then you could get away with just 
utils . network_f uncs. 

O filesystem path: An alternative way is to specify the filesystem path to the leafnode. This 
may allow you to run leafnodes even when they're not in leafbags (caveat emptor). To 
do this, you just give the filesystem path to the python file containing the leafnode, 
followed by the name of the leafnode in the file, with an '@' in between. For example: 
leaf bags/my_leafbag/utils/network_f uncs . py@ping. As with the 
python import path, if you put the MainLeaf decorator on the ping leafnode, you can 
leave off the @ping component. 

• host_spec: This is a specification of the host to connect to with palantir. Since palantir 
currently only runs over TCP/IP, this must be either the IP address or hostname of the host. 

• args_and_kwargs: Here you specify, space-delimited, the arguments and keyword 
arguments to the leafnode. See the output of bin/undermine -h for an extensive 
description of exactly how arguments and keyword arguments are specified. 

When undermine runs it logs into two files: script.log and undermine.log. script.log contains logging and 
output specifically from the test script, undermine.log contains lower-level logging from undermine and 
palantir, logging which the test script developer may not care about. These files are located in the script's 
output directory. 

When running test scripts standalone with undermine, by default, output directories are stored under 
OUtput/undermine/<USERNAME> relative to the tybase root, where <USERNAME> is the user 
name of the user running undermine. Under this directory, a subdirectory named with the current date 
and time is created, and this timestamp directory is the output directory of the leafnode. Also, in the 
<USERNAME> directory will be a symlink called “latest” which always points to the most-recently-run 
leafnode. This is especially helpful when there are lots of output directories sitting around in a 
<USERNAME> directory. 

Batch Testing 

The plundermine tool provided in tybase allows running simple combinations of tests against specific 
test resources. To use it, you give plundermine a leafnode to run, and lists of hosts and parameters for 


27 


UNCLASSIFIED 


UNCLASSIFIED 


each host and parameter slot the leafnode takes. Plundermine will generate all the possible 
combinations and run them with a level of parallelism (up to a configurable maximum number of 
concurrent undermine runs). In the fairly common degenerate case of a leafnode which accepts only 
one host and no arguments, plundermine is an effective tool for running a given leafnode against a 
whole set of resources. This is useful for some range management tasks. 

See the output of bin/plundermine -h for full details. Some examples are: 

• Run a leafnode which only accepts one host against the three listed hosts: 

bin/plundermine underlib . test_leaf node 

192 . 168 .56.1, 192 . 168 .56.2, 192 . 168 .56.3 

• Run a leafnode which accepts two hosts and no arguments against all possible combinations of 
hosts listed in the two specified tiles: 

bin/plundermine underlib . client_server_test file: clients 
file : servers 

• Run a leafnode which accepts two hosts and two arguments against all possible combinations of 
the hosts from the first tile, the hosts from the comma-separated list for the second host slot, 
and the specified values for the two argument slots: 

bin/plundermine underlib . complex_test file : first_hosts 
192.168.56.1,192.168.56.2 - one, two, three x, y, z 

For plundermine, output directories are stored in output/plundermine/<USERNAME- 
TIMESTAMP>, where <USERNAME> is the name of the user who ran plundermine, and 
<TIMESTAMP> is the date and time at which plundermine was run. Inside each of these directories 
are numbered subdirectories representing each of the test instances run. These numbered 
subdirectories are each undermine output directories containing the undermine log and data tiles. 

Test Plans 

Test plans are the units of work performed by Overmind. They specify what test to run (the leafnode the 
user already has) and what to run it on ("all versions of Windows", "all languages of Windows XP SP2", 
etc). Overmind utilizes the reaper to revert assets to previously stored state (typically, reverting to a 
snapshot on a VM). Overmind stores the results of undermine runs in a database which can then be 
viewed through the overview web gui. Like leafnodes, test plans are written in python and stored in 
leafbags and are referenced on the command line in the same manner (either as python import paths or 
filesystem paths). 

Test Plan Concepts 

The following terms will be useful to know when writing and running test plans: 

• test plan: A python script which defines test cases. This is the thing you run with overmind. 

• test case: A description of what leafnode to run, on what assets, with what parameters. 

• combo or test instance: A specific combination of leafnode, assets and parameters. A test case 
expands into multiple test instances at run time. These test instances represent individual runs 
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of undermine. NOTE: In overview, the overmind web gui, test instances are referred to as test 
cases. 

• namespace: An overall identifier in overmind under which multiple plans can run. 

• purge: To immediately cancel a running namespace, plan, or test instance. 

• reap: To revert an asset to a previously stored state. 

• recipe: A definition of an OS which can be placed on a computer (e.g. the family, service pack, 
architecture, language, installed apps) 

• computer: A computer on which a recipe can be installed (e.g. a VM or physical machine) 

• resource: A specific computer with a specific recipe on it. 

Example Test Plan 

In your tyworkflow repository, look at 

src/leaf bag/overlib/preflight/service_ping_plan . py. This is a test plan which runs 
the service_ping_test on all unique combinations of computer and recipe in the range, which for 
our purposes is equivalent to all the resources on the range. We’ll walk through this line-by-line: 

from tyworkflow. support . planlang import * 

This line imports the plan language objects used in writing the test plan 

test = TESTCASE ( 

We begin the definition of a test case which will be used to generate all the actual tests run. 

script = ' overlib . preflight . service_ping_test ' , 

This defines what leafnode to run. You must use the “python import path” method of specifying the 
leafnode to run; do not use a filesystem path. 

hostslots = [HOST ( ) % FACTORS(computer_id=l, recipe_id=l) ] , 
This defines how to generate the actual tests run, or “combos”. This hostslots setting indicates the test 
only takes one host (since a list of only one element is provided), puts no constraints on the chosen host 
(because of the empty argument list to HOST), and indicates that the combination of computerjd and 
recipejd for each chosen host must be unique. This is covered in-depth later. 

samples = -l, 

This line specifies how many of the generated combos to actually choose. The special value -1 indicates 
that all combos should be used. 

namespace = ' preflight-ping-$t 1 , 

This specifies the name of the namespace to use if no namespace is . 

The values defined in the TESTCASE declaration may be overridden on the command line. 

EXECUTE ( 

testcase = test, 

) 

This block sets the testcase defined above to be execute. 
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Parsing Test Plans 

Prior to actually running a test plan, there are a couple operations which may be performed on it to 
verify that it will do what you expect. Because you configured your database by editing rc/db . rc in 
tyworkflow earlier, you can run these steps locally even though your test plans will be run remotely. 

The process_plan command provided in tyworkflow provides the parse and solve subcommands. 
Parse will parse your test plan and give back to you overmind’s understanding of what you’re written. 

You can use it as a quick verification that you wrote what you intended. To parse the example plan from 
above, you would run 

bin/process_plan parse overlib . preflight . service_ping_plan 

If you wanted to parse the plan but then override the samples setting, you could run 

bin/process_plan parse overlib. preflight . service_ping_plan samples=10 
and you would see that change reflected in the output. 

The solve subcommand will parse your plan and then show you what combos your plan would generate. 
To solve the example plan, you’d run: 

bin/process_plan solve overlib . preflight . service_ping_plan 
You could override samples like above, and depending upon how diverse your range is, you may see the 
number of combos decrease. 

Running Test Plans 

The remote commit command (bin/remote_commit) in tyworkflow allows you to submit your tests 
to a remote test server. Once your test is submitted, you can continue your development in your local 
environment without affecting the test you just submitted. You can even submit tests in parallel, 
allowing you to try one approach to solving a problem, submit a test of it, then try a different approach 
and submit a test for that approach in a different namespace. Here we explain how to run test plans, but 
remote commit has more functionality which is covered in detail later on. 

Test plans are run though remote commit with either the run or r unlite subcommands. These 
commands sync your local environment up to the test server and submit your test to your remote 
overmind instance. These commands take the same arguments as process_plan’s solve and 
parse. To submit the service_ping_plan with remote_commit, you’d run: 

bin/remote_commit run overlib . preflight . service_ping 

If you wanted to limit the number of samples, you would run 

bin/remote_commit run overlib . preflight . service_ping samples=10 

Another useful option is to specify some notes on the namespace with the n_notes argument: 

bin/remote_commit run overlib . preflight . service_ping samples=10 
n_notes="please work" 

Working with a Range 

As a developer, overview will be your primary interface to range testing. Overview lets you browse test 
results and reserve machines for use outside of normal automated testing. 
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Seeing Test Results 

To see test results, navigate to overview’s Namespaces page (e.g. 

http://testserver.example.com/test_namespaces.php). Here you’ll see a listing of all the namespaces 
that have been run. When remote commit is in use, there will usually be only one plan in a namespace. 

At each of the list levels, on the right side of the table, you’ll see a summary of how many testcases in 
that entity are in the various states a test can be in (either pending, running, or one of the completion 
statuses). 

The list pages also have forms at the top allowing you to filter by matching on various attributes of 
namespaces, plans, or test cases (specified by naming the field of the table to filter on and the pattern to 
match, e.g. status=error to see all error testcases or n_name=~pref light to see all 
namespaces whose name matches the pattern “preflight”). 

The Refresh field specifies the interval in which any particular page will refresh. This is particularly 
useful for monitoring test results as they finish. 

Finally, the Limit field allows you to specify how many rows you want to be displayed (whether 
namespaces, test plans, or test cases). This prevents loading an entire page with 1000’s of rows if the 
loading time would take too long. 
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NS Notes: 

o 


Test Plan: 


Filter: n_name-= ' fatl 

Test Case: 

(*) c: 

Refresh: | | secs 1 START l) 



- 



start time 

finished 

ir 


1“ 

hIhhuhiih 

[*| 

408 

JacalDettaTest 

34 Finished 

2013-09-17 12:5420 EDT 

34/34 

34 



c 

• 

407 

DdtaTest 

17 Finished 

26 Finished 

2013-09-17 1237:50 EDT 
|2013-09-16 0823:49 EDT 

17/ 17 

26/26 

1“ 

26 



HI 

405 

QsInfoTest 

60 Finished||2013-09-15 16:10:24 EDT 

60/60 

60 




Clicking on a namespace brings you a list of all of the test plans in that namespace. 


Test Plans (test cases) 



Clicking on a plan name brings you to a list of all the test cases in the plan. 
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Test Case Details 


Namespace: 

OsIrfoTest ||405| 

• 

1 results 

NS Notes: 


Test Plan: 

verify osinfo plan-2013 09 17-20 53 40 || 

421 

Test Case: 

verify_osinfo_plan-2013_09_17-20_53_49_955046-421 

616 


Result Code: m 

Start: 2013-09-17 16:53:50 EDT 


End: 2013-09-17 17:07:03 EDT 


Script: underHb.preffight verify_osinfo_test 
Host_0: 1 72.2.2. 122/windows:2003 R2;0;zh-T\V;X86,palantir;vm 


Result: patch_level:spl!=spO 

Output Dir: proj tjTant-de\- output OsInfoTest verify_osinfo_plan-2013_09_17-20_53_40'verify_osinfo_plan-2013_09_17-20_53_49_955046-421 
Undermine Logs run task log trawl 1756 

script-log trawl 981 


udala-call-00000001.dat (ts * 1 144 metajraw) 314 

udata-result-00000001.dat fraw} 20 meta fraw] 311 

udata-result _ code-0000000 1 .dat {raw) 4meta(raw}326 
ndata-timps-00000001 riat trawl 87 meta trawl 315 

undermine log trawl 14976 


HostO 

win2k3r2std-sp0-x86-tv 

172.2.2.122 


The file script .log contains logging output specifically written by the test script developer (by calling 
self . log or host . log in a test script) which undermine . log gives lower-level undermine 
framework logging information. See these files in the case of errors or unexpected results with your 
tests scripts. 

Clicking on any of the log hies leads to the raw data from the file system. 
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/proj/tyrant-dev/output/OsInfoTest/verify_osiiifo_plan-2013_09_17-20_53_40/verify_osinfo_plan-2013_09_17- 

20_53_49_952576-421/script.log 

RegExp Filter: I B 


Test Plans in Depth 

Within the previously outlined basic structure of plan files, there are many constructs in the "planlang” 
(provided in your plans by the line 

from tywor kf low. support . planlang import * 
present at the top of each plan file). Together, these constructs are used to build the specification of 
what tests to run, with what arguments and what types of test resources. 

FILTER 

This is a base class which supports abstract filtering based on values of named attributes. It is extended 
by some other classes in the planlang and is generally not used directly. The typical usage is the HOST 
subclass. 

HOST 

A FILTER subclass that filters resources based on attribute value constraints. The value can be a 
singleton or a list of values. The constraints are specified in the constructor with the general form: 

HOST(<attr>=<value> | [<value>, <value>, . . . ] , <attr>=. . . ) 

For example, 

HOST (family^ 1 windows' , os=['2k', ' xp 1 ] , ossp='sp0') 

HOST objects are the primary means to define the desired set of resources to use for a given test script. 
HOST objects are used in the host slots argument to the TESTCASE constructor, explained later. 

IMPORTANT: The values for "family” and “os” and all of the other fields on which a user can write a 
filter are set by the Overmind database. These values can be viewed using Overview’s "Recipes” and 
“Resources” pages to determine what valid values are. For example, if the administrator sets the "os” of 
a box to be the string “xp_pro” instead of “xppro” and the user wants to run on XP boxes, the filter for 
“os” needs to be "xp_pro” - the exact string match. This loosely-defined schema is nice for rapidly 
adapting to new recipes; however, it does require coordination between the users and the 
administrators. 

We recommend the schema style specified in the recipes . CSV file in the docs/ directory of 
tyworkflow; however, it is more important that an organization is simply be consistent with whatever 
schema they choose. 
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FACTORS 

The class used to control the sampling of test instances. For example, consider a plan with 3 host slots 
and a resource pool of 100 machines. In the worst case, this could generate 100 A 3 potential test 
instances. If each test takes 10 minutes, even with max parallelism of 33 simultaneous test instances, it 
would take a minimum of 10,000,000/33 minutes or ~21 days. In this case, you may want to sample the 
set. FACTORS objects specify the attributes of interest to vary across test instances. Other attributes will 
be randomly selected based on resource availability. The constructor defines the attributes of interest 
with the general form: 

FACT0R(<attr>=True | False, ...) 

For example, 

FACT0R(family=True, os=True, ossp=True, lang=True) 

TESTCASE 

The class used to compose HOST, FACTOR, and parameter values to form a specification for a set of test 
instances. Specifically, a TESTCASE constructor takes: 

• script: Name of test script (leafnode). 

• hostslots: List of FILTER objects, defining the number of host resources and their 
constraints. For example: 

[H0ST(), HOST ( ) ] 

The number of elements in the hostslots list defines the number of resources each testcase 
will use and should match the number of resources the test script defined in the script argument 
requires. 

• paramslots: List of parameter values (defined as a list). For example: 

[['a', 'b'], [ 'C ]] 

Combos are generated for each potential value of a parameter slot. In the given example, the 
first parameter slot can be either ‘a’ or ‘b’, but the second parameter slot will always be ‘c’. So, 
for a very simple example range with only one test resource, a plan with this paramslots 
setting would generate two combos, one with arguments ‘a’ and ‘c’, the other with arguments ‘b’ 
and ‘c’. 

• filter: Singleton FILTER object that defines a global constraint over resources. For example: 

HOST ( pool= 1 pname ' ) 

• xattrs: Singleton XATTRS object, defining attribute constraints across host objects. For 
example: 

XATTRS ( vlan= ' same ' ) 

This line would ensure that the hosts are on the same subnet. 

• factors: Singleton FACTORS object, defining sampling attributes. For example: 

FACTORS (0S=1) 

• samples: Maximum number of sample test instances to run. 

• replications: Number of times to run each sample. 

• priority: Numeric priority of test instances (used to sort scheduling queue). 

• namespace: Namespace name to use when storing test results in database. 

• post_ops: List of functions to run as post operations. 
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• n_notes: Informational notes to store with the namespace this testcase will run in. Remember, 
this MUST be quoted if it contains spaces. 

• p_notes: Informational notes to store with the test plan this testcase will run in. Remember, 
this MUST be quoted if it contains spaces. 

EXECUTE 

The class used to define which TESTCASEs to run for the plan file. The separation of TESTCASE and 
EXECUTE allows you to compose TESTCASE separately from specifying which ones to run. The 
EXECUTE constructor takes the TESTCASE object and an optional set of keyword arguments. If the 
optional TESTCASE keyword arguments are provided, they are used to override the value for the given 
TESTCASE. Note this is purely for convenience and EXECUTE( testcase, **plan_ops) is 
equivalent to: 

EXECUTE( testcase / TESTCASE( **plan_ops) ) 


PARSE 

Returns the list of EXECUTE'd TESTCASEs for a given plan file. Any list operator can be applied to this 
list for composing plans from plans. As with the EXECUTE statement optional arguments can be 
provided to override the value for the resulting TESTCASEs. Note this is purely for convenience and 
PARSE(plan_file, **plan_ops) is equivalent to: 

map (lambda x: x/TESTCASE( **plan_ops) , PARSE ( plan_file ) ) 

Planlang Operators 

The language provides a set of operators over objects for plan reuse and simplification. For FILTER 
operators it is best to think of a FILTER object as representing a set, specifically, the set of resources 
that satisfy the constraints. The valid operators are: 


• FILTER & 

• FILTER | 

• FILTER - 

• FILTER % 

• FILTER % 

• TESTCASE 


FILTER: A new FILTER object representing the set intersection of the two. 
FILTER: A new FILTER object representing the set union of the two. 
FILTER: A new FILTER object representing the set subtraction of the two. 
FACTORS: Binds the FACTORS to the FILTER object 
XATTRS: Binds the XATTRS to the FILTER object 

/ TESTCASE: Copy of left TESTCASE with attributes overridden with the 


defined attributes of the right TESTCASE (undefined attributes use the value from the left 
TESTCASE). 

TESTCASE & FILTER: Copy of TESTCASE with global filter constraint intersected with 
FILTER. 

TESTCASE | FILTER: Copy of TESTCASE with global filter constraint unioned with 
FILTER. 

TESTCASE - FILTER: Copy of TESTCASE with global filter subtracted with FILTER. 


Plans of Plans 

One incredibly useful feature is to generate plans of plans. A typical use case is to have a high level plan 
that kicks off a lot more lower level plans. For example, a regression_test plan could include kicking off 
all of the relevant plans for a single situation. 
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To do this, one needs to generate a plan file that calls run_plan on other plans. The arguments to 
run_plan are the following: 

namespace: typically globals() 

search_path: typically the leaf bag containing the plans 

plan_narne: specific plan name to be called by this plan (without “.py” extension) 

maxCount: how many times to run this plan (equivalent to setting “samples=” from the 

remote_commit command line 

For example, 

from tywor kf low. support . planlakng import run_plan 

run_plan(globals( ), "my . leaf bag", "MyPlan", 5) 
run_plan(globals( ), "my . leaf bag", "MyPlan2", 2) 

NOTE: Any command line arguments (e.g. “samples=”) passed to remote_commit will override any 
arguments passed to the test plans themselves via run_plan. 


Remote Commit in Depth 

Remote commit is provided as an alternative method of setting up an overmind test range and 
submitting plans to it which is useful for environments of multiple developers working against a single 
overmind-controlled test range. 

For background, the standard (non-remote-commit) way of doing things is to, on a single machine, run 
the overmind, reaper, and overview servers. The user may choose to set up a MySQL database for 
overmind to store information about assets and test results in, or they may use the default SQLite 
database. The user then links in their testing code as a leafbag in their tybase repository and submits 
plans from their tyworkflow repository (which has tybase linked in at media/tybase). This lends itself 
well to a single developer working on the range, but doesn't work so well for multiple developers on 
different workstations. Under this model, they would have to SSH in to the machine running overmind, 
cd to the appropriate tyrant directory, put their testing code in place, and then submit a plan. Parallel 
work by multiple developers is not favored in this model. 

In contrast, remote commit gets around these problems. With remote commit, a single MySQL database 
serves as the point of concurrency. Global instances of reaper and overview are run on a server and a 
directory is created on this server to serve as the root for remote committing to. Users set up their own 
local working directories of tybase, tyworkflow, and their SUT(s). When the user submits a test plan with 
remote commit, remote commit rsyncs all their testing code up to the server (in a subdirectory for the 
developer) and runs the overmind in the tyworkflow the user synced up to schedule the user's tests 
(referring to the database to see what's available). The results of the user's tests are recorded to the 
global MySQL database, which all the users can view on the global overview instance. Any files that 
resulted from the test are also stored on disk and accessible from the overview interface. Once the user 
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has run remote commit, which only takes as long as is needed to sync their code up to the server, they 
user may continue development in their working directory without affecting the tests that are running 
on the server. By specifying alternate usernames when submitting test plans, the user may even submit 
one test plan while another is already running. 

SUT Preparation 

In order to use a given SUT with remote commit, you must prepare a shell script that provides some 
functions and constants which remote commit will use. These should be placed in a file named 
rcoverrides.sh in the root of your SUT's leafbag. It is important that the functions respect posix 
conventions for return code, i.e. return 0 on success, nonzero on error. All override functions run relative 
to the tyworkflow root. Likely, you'll want some of your functions to run relative to the tybase root. In 
that case, you'll need to do something like pushd media/tybase at the top of your function and 
then popd at the end. 

Functions: 


• _rcoverride_build: Performs any steps necessary to build the SUT or prepare it for testing 
that are not encompassed in the actual testing code. For example, if your SUT is a single C file 
that requires compilation before testing, your _rcover ride_build function might just run 
gcc. If your project is more complicated and has a makefile, your _rcoverride_build could 
run make. 

• _rcover ride_clobber: Called immediately after running a make clobber in 
tyworkflow’s root, as part of the clobber command. Performs a thorough cleanup of the SUT 
directory. Continuing on the example from the previous function, the 
_rcoverride_clobber for a simple one-file C SUT might just delete the binary resulting 
from the compilation encoded in _rcoverride_build, while the implementation for a 
project with a makefile might run make clean. 

• _rcoverride_Stop: Called when running the stop command, immediately after shutting 
down the overmind service. 

• _rcoverride_summary: Called when running the summary command, just after running 
bin/scan_output (which summarizes the undermine runs recorded in the output directory. 
This function allows you to add any custom output to the summary output. 

• _rcover ride_submit: Called during running of the submit command on the testing server, 
immediately prior to actually submitting the desired plan to the running overmind. This allows 
you to do things like make settings changes to the running overmind conditionally based on 
which test is being run (a specific example would be to increase the maximum number of 
children for certain larger test plans). 

Constants: 

• UNDERMINES: Overrides the default maximum number of children your remote overmind 
process will allow. 
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• CLIENT_TIMEOUT: Overrides the default client timeout (maximum running time for various 
interactions with overmind). 

Leafbags with non-leafbag dependencies 

In order for tests to work with remote commit, the tests and all their dependencies must be linked in to 
your environment so that they will all end up on the testing server during remote commit's rsync. For the 
case of a self-contained leafbag, this works trivially: your SUT's leafbag is linked in to your tybase 
repository (in leafbags/), so when remote commit syncs, the leafbag is synced up. When your 
leafbag has non-leafbag dependencies, those dependencies must also be linked in to your copy of tyrant 
so that they will also be synced up during remote commit, and your testing code must be written so that 
it can reference the dependency relative to the tybase root (so that there are no hardcoded paths that 
will break when the entire directory is synced to some other system). While there is no way this 
technically must be done, convention is to link in your extra dependency with a symlink in tybase’s 
media directory, then write your testing code to refer to the supporting components in that location. 
Then, using the search_media_path function provided by in tybase . support . util, you can 
retrieve the path to the directory you linked in and then do whatever you need to with it (e.g. open a file 
relative to it, add it to the python path so you can import from it, etc). Call search_media_path with 
the name of the symlink you created inside of media, and it will return a usable path to your linked-in 
media. 

Example 

Suppose you have a project with a repository called eproj which contains the following two 
subdirectories (among others): leafbag (the leafbag with leafnodes for your project) and data (a 
directory with some sort of supporting files that are used both for your tests scripts in leafbag and by 
other parts of the software). You would link the leafbag in to tyrant's leafbags/ as always. Then, you 
would also put a symlink in tyrant's media/ pointing to eproj's data subdirectory. For example, 
assuming your current working directory is the root of your tyrant copy and eproj is a sibling of your 
tyrant copy, you could run 

In -s . . / . . /eproj/data media/eproj -data 
Then, a hypothetical eproj test script might contain code like the following to make use of files in that 
directory: 

from tybase . support . util import search_media_path 
epdata = search_media_path( ' eproj -data' ) 

with open(os . path . join(epdata, 'seed-00l.txt'), ' r b ' ) as seedfh: 
seed_dat = seedfh . read ( ) 

Shared directories 

It may be that your testing involves some large set of files which don't change very much and can be 
shared among developers. While each developer does need their own copy of these files on their local 
workstation for any local tests they might be doing, you would rather not have multiple copies of this 
large set of files on the testing server (since each developer has their own subdirectory of the commits 
directory to which they would have to sync their own separate copy of the shared files) and would rather 
not have to go through syncing those files every time a developer runs a test. 
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Remote commit provides a way to handle this. In the remote_COmmit . rc file, create a section called 
shared_dirs. Each option in this section defines a single shared directory. The name of the option is 
the path to the shared directory relative to the root of tybase (i.e. where the shared directory currently 
resides in the tybase clone in your local testing environment). The value of the option is the path to the 
actual shared directory on the testing server. When you do a remote commit operation that involves a 
sync (run, runlite, sync, or synclite), the paths given as the option names in the 
shared_dir s section are excluded from the sync. After the rsync part of the sync process is complete, 
remote commit will create symlinks inside the remote tybase root (at the paths given as the option 
names) pointing to the paths given as the values to those option names. This saves having multiple 
copies of the large shared files on the testing server, however it is your responsiblity to make sure the 
copy of the shared directory on the testing server is kept up to date, since that will not happen 
automatically when a developer does a "sync”. 

Example 

Suppose your eproj test scripts require a set of installers and data files that exist in a directory called 
epro j_data. Currently, you make this set of files accessible by putting a symlink named 
eproj_data in tybase's media directory which points to the actual eproj_data directory (so the 
location of that symlink relative to the root of tybase is media/eproj_data) and then writing your 
test scripts to access the files out of media. To set this up as a shared directory for remote commit, you 
would do the following: 

• Place a copy of the shared directory somewhere on the testing server. For our example, we'll put 
it at /proj/eproj_data. 

• Add an option to the shared_dirs section of ty workflow’s rc/remote_commit . rc 
whose name is the location of the shared directory inside tybase and whose value is the location 
of the shared directory on the testing server. For our example, this is: 

media/eproj_data = /proj/eproj_data 

• Now, when you do a sync, media/eproj_data will be excluded from the initial rsync 
operation. Then, after that operation completes, a symlink will be created inside the remote 
tybase at media/eproj_data pointing to /proj/eproj_data. 


Usage 

Remote commit provides several commands via the bin/ remote_commit script (in tyworkflow) 
which are used to sync your SUT up to the testing server, run tests, and administer your remote instance 
of overmind. These commands are explained below grouped by use case 

(Dry)Running Tests (run, runlite, submit, num_combos) 

The run and runlite commands are the backbone of running remote commit; you can ignore all the 
others and still work effectively with these two commands. They are basically wrappers which run 
several commands under the hood. They both sync your testing environment up to the testing server, 
start your remote instance of overmind, and then submit the given plan. The difference between the lite 
and full versions is that the full version does a clobber and build on the local side before syncing up to 
the testing server, whereas the lite version does not do this. 
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Usage: 

bin/remote_commit [-u <USER>] run <PLAN> [<process_plan_opts>] 
bin/remote_commit [-u <USER>] runlite <PLAN> 

[<process_plan_opts>] 

where <PLAN> is the designator for a plan file (either a python import path or filesystem path). The - u 
option specifies the subdirectory of the commits directory to work out of, and defaults to the current 
username (of the person running bin/remote_COmmit). In the case of run, the developer's testing 
environment will be synced up to the named subdirectory of the commits directory, the overmind in that 
directory will be used, etc. 

Also, since run and runlite end up calling bin/process_plan on the remote side, you may 
override certain testcase attributes just as you would if you were calling bin/process_plan directly. 
This ability to override testcase attributes is available for any of the remote_commit commands that call 
bin/process_plan on the remote side. The one we find most useful is to override the samples 
value to run a subset of a potentially large set of combos. For example, your testplan may generate 100 
combos, but you only want to run a random ten of them. Then, you would do: 
bin/remote_commit run PLAN samples=10 

The submit command simply syncs up your testing environment and submits the given plan without 
running the build step. This is useful if you make local changes ONLY to your SUT’s testing code (or other 
minor local changes which don’t require rebuilding your SUT), because in that case the changes you're 
syncing up won't have any effect on your remote overmind instance, so a restart is not necessary. If, 
however, you’ve made SUT changes that require a rebuild, then submit may not be safe to run; you 
should use run and runlite instead. If you’ve made changes to actual tyrant code, then you need to 
do the sync command (explained later) first to force your remote overmind instance to restart. 

Usage: 

bin/remote_commit [-u <USER>] submit <PLAN> [<process_plan_opts>] 

The num_combos command allows you to see how many combos your test plan will run when 
submitted. This is analogous to the bin/process_plan solve command used with the classical 
overmind setup. 

Usage: 

bin/remote_commit [-u <USER>] num_combos <PLAN> 
[<process_plan_opts>] 

Syncing your testing environment (sync, synclite, diff) 

These two commands sync your testing environment up to the testing server. As with run and runlite, 
the full version does a local clobber and build, the lite version does not. An added difference between 
sync and synclite is that sync forces your remote overmind instance to restart, whereas 
synclite does not. If you make changes to core tyrant code and want that to take effect on the 
remote side, you need to use sync, since if the remote overmind doesn’t restart, your changes may not 
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take effect, depending upon what you changed. Generally, developers and testers won’t be making 
changes to overmind, but if you receive an updated delivery of tyrant code or the administrator makes 
some changes, you may need to run a full sync. 

Usage: 

bin/remote_commit [-u <USER>] sync 
bin/remote_commit [-u <USER>] synclite 

The dif f command is provided as a dry run of syncing. It just uses rsync's dry run capability to show 
you what files will be uploaded/changed/deleted when syncing to the testing server. 

Usage: 

bin/remote_commit [-u <USER>] diff 

Administering your remote overmind instance (start, stop, restart, set_children, get_children) 

To start, stop, or restart your remote overmind instance, the respective commands are provided. Users 
do not typically use these since they are handled automatically when running run or sync commands. 
During the stop command, the custom _rcover ride_Stop function is run, if provided. 

Usage: 

bin/remote_commit [-u <USER>] start 
bin/remote_commit [-u <USER>] stop 
bin/remote_commit [-u <USER>] restart 

The set_child r en command allows you to set the maximum number of undermine processes your 
remote overmind process will run in parallel. 

Usage: 

bin/remote_commit [-u <USER>] set_children NUM_CHILDREN 
where NUM_CHILDREN is an integer telling how many children to run in parallel. 

The get_child r en command tells you the current max number of undermine processes. 

Usage: 

bin/remote_commit [-u <USER>] get_children 

Building and clobbering your SUT (build, clobber, rclobber) 

The build and clobber commands run the _rcoverride_build and 

_rcoverride_clobber functions you provide in your rcover rides . sh file; in other words, they 
locally build and clobber your SUT. The rclobber command runs the _rcover ride_clobber 
function, but on the remote testing environment. 

Usage: 

bin/remote_commit [-u <USER>] build 
bin/remote_commit [-u <USER>] clobber 
bin/remote_commit [-u <USER>] rclobber 
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Seeing results (summary) 

In addition to viewing results of tests via the overview GUI, you can also use remote commit's summary 
command, which prints out a summary of your remote testing environment's output directory. If 
provided, the _rcover ride_summary function is also run after printing the default summary 
information. 

Usage: 

bin/remote_commit [-u <USER>] summary 

Other commands (client) 

- u <USER>The client command runs an arbitrary command with the overmind client 
(bin/overmind_admin) on the remote overmind instance. 

Usage: 

bin/remote_commit [-u <USER>] client <CMD> 
where <CMD> is the command you want to run. See bin/overmind_admin - h for a list of potential 
commands to run. 
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Appendix A - Event Detection 

Via the magnum add-on repository, Tyrant provides the ability to run arbitrary test scripts in a wrapper 
which monitors a test resource’s screen for changes. 

Magnum provides two methods of acquiring screenshots: using native Windows functionality to take 
screenshots (which only works for Windows resources and can be affected by conditions on the resource 
being tested, but can work on VMs and physical machines alike) and using ESXi screenshot functionality 
(which only works for VMs, but works for all OS families and is unaffected by conditions on the resource 
being tested). Here, we cover how to setup and verify ESXi-based event detection. 

This appending assumes that the test range you’re using has already been set up for event detection. 

Event Detection Theory 

Event detection works by taking screenshots according to a configurable interval and comparing them to 
find differences. When differences are found, they are analyzed to determine whether they are 
considered significant or not. A difference is significant if the number of changed pixels in a set of 
predefined areas of interest exceeds a defined threshold. The areas of interest and the threshold were 

determined empirically and are defined in src/magnum/event_detectors/ init . py as 

percentage boxes bounding the areas of interest. The current threshold is 10000 pixels, and the current 
areas of interest are: 

• A box in the bottom right corner of the screen, extending 25% toward the left and 50% toward 
the top. 

• A box in the center of the screen, whose edges are all 30% away from their respective screen 
edge (i.e. the left edge of the box is 30% from the left edge of the screen, the bottom edge of the 
box is 30% from the bottom edge of the screen, etc). 

• A box in the upper right corner extending 20% toward the left and 20% toward the bottom. 

When no problems occur with event detection, the event detection harness simply returns the result 
returned by the underlying test script. If, however, the test script returns SUCCESS, but any problems 
are encountered with event detection (e.g. not being able to take screenshots frequently enough to 
satisfy the configured interval), then the harness will return ATTENTION along with a message 
describing what happened. 

Testing in Adverse Environments 

The goal of testing is always to determine truthful outcomes to potential scenarios as early as possible, 
so that risks can be understood and evaluated. It is critical for users and testers to be aware of the fact 
that testing of any kind produces traces and artifacts. These traces and artifacts are created because it is 
impossible to actuate components that set up test preconditions without changing the state of the 
machine under test. There are many ways to achieve these actuations. Different methods can (and 
sometimes do) provide different results. This is especially true in adverse environments. For example: a 
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tool that passes when a user runs the program from the desktop with the mouse could fail or cause a 
pop-up if started by another process. 

Automated testing frameworks like DART allow testers to cover a greater number of potential scenarios 
than they could manually. This creates more confidence in the tools being tested. However, it is critical 
to understand that the automated framework runs in a formulaic way, so it is possible that the methods 
chosen could routinely produce different results in the real world. It is even possible that by allowing the 
automated framework to run in an adverse environment, that environment will be changed enough that 
a different result could show up. 

Tester note: You should run a statistically relevant subset of the tests by hand to verify the results given 
by the automated framework. Follow the spirit of the test plan and ensure that doing things manually 
produces the same results. This will nearly always be the case, but we have observed instances where 
there are slight deviations in the past. 

Environment Setup 

In order to use event detection, you need to do some setup in your local testing environment (the 
environment in which you run remote_commit to submit tests to the range). 

• In the same directory where your tyworkflow and tybase clones are, clone the magnum 
repository (hg clone http : //testserver . example . com : 8000/magnum). 

• In that same directory, also clone the provided PIL (Python Imaging Library) repository matching 
the architecture of the test server. For example, if your test server is running 32-bit Linux, 
choose the “PIL-linux-i686” repository, but if it’s running 64-bit Linux, use “PIL-linux-x86_64”. 

This library is used to compare screenshots to find changes. 

• In your magnum clone, copy conf ig/main . conf . example to conf ig/main . conf and 
set the following settings. Magnum has many features besides event detection, so some of 
these settings are unrelated but need to have some value set for them to prevent warning 
messages. 

o Set tester/ evdet_type to esxi. 

o Set tester/esxi_evdet_ds_name to the name of the NFS datastore you created 
in the previous section (e.g. tyrantshare). 

o Set tester/esxi_evdet_local_ds to the path on the test server of the directory 
what was exported via NFS in the previous section (e.g. 

/proj /testing/tyrant share), 
o Set server/upd_root and server/inst_root both to /tmp. 
o Set server/ip_addr to 127.0.0.1. 
o Set repository/repo_path, repository/repo_url, 

repository/repo_user, and repository/repo_pass to UNUSED. 
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Usage 

To use event detection, you use an event detection harness leafnode provided by magnum. You tell it 
what test script you want it to run and what arguments and keyword arguments to run it with and give it 
various other pieces of information to configure the event detection. The harness takes the following 
arguments: 

• host_index: Zero-counting integer index of which host event detection should be performed 
on. This is necessary when your test case uses more than one host, but you want event 
detection run on some host other than the first. Default is to use the first host. 

• interval: Float number of seconds for the screen polling interval. Default is 5 seconds. 

• use_emissary: Boolean indicating whether or not to use emissary when using the onhost 
event detection method. Not applicable for esxi event detection. Default is False. 

• type: Selects which type of event detection to perform (esxi or onhost). Default is 
onhost. For what this appendix is covering, you will always choose esxi here. 

• esxi_host: For esxi event detection, specifies the ESXi server to connect to to take 
screenshots. This may be either the specific ESXi host the VM resides on, or (we assume, but 
have not tested) a vCenter server for the range in which the VM resides (we assume this because 
other operations like reverting work both when directly connected to an ESXi host or when 
connected to a vCenter server). If not specified, the harness will attempt to query the overmind 
database (if present) and will assume the reaper field for the test resource indicates the DNS 
name or IP address of the ESXi host the VM resides on. 

• esxi_user: Username to use for connecting to the ESXi host. 

• esxi_pass: Password to use for connecting to the ESXi host. 

• vm_name: ESXi name of the VM. If not specified, the harness will attempt to query the 
overmind database (if present) and will use the computer name field as the VM name in ESXi. 

• debug: Boolean indicating whether or not to perform event detection debugging. This causes 
the generation of extra log messages and several intermediate images during the screen 
differencing process. Do not use this unless you actually need to debug the screen differencing 
process. This does not affect debugging for the test script being wrapped. Default is False. 

• debug_dir: If debug is True, specifies a directory path to which to output the extra files 
generated by debugging. Default is to use a subdirectory of the output directory for the run of 
the harness. 

• keep: Boolean indicating whether or not to keep screenshots which indicate differences 
determined to be insignificant. Default is False. 

• test: Specification of the leafnode to run. Specify this as a “python import path”, not a 
filesystem path. 

• test_args: List of positional arguments to the specified leafnode. 

• test_kwargs: Diet of keyword arguments to the specified leafnode. 

Note that these arguments are subject to the argument quoting rules of undermine. See the output of 
bin/undermine -h (run in tybase) for details. 
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With Undermine 

To use event detection to run single test instances with undermine, simply call the event detection 
harness with the proper arguments. For example: 

bin/undermine magnum . harness . evdet_harness 192.168.56.101 
192.168.56.103 192.168.56.104 -- host_index=@l interval=@3 . 0 
type=esxi esxi_host=192 . 168 . 56 . 10 esxi_user=someuser 
esxi_pass=somepass vm_name=test_vm_0Ol 
test=mysut . tests . sometest test_args=@"[yes, yes, no]" 
test_kwargs=@"{one=l, two=2}" 

If running out of an environment linked to an overmind range (i.e. tybase and tyworkflow are linked and 
tyworkflow’s db . rc is configured to use an overmind database) and using IP addresses that are part of 
the range, you can leave out the esxi_host and vm_name arguments and they will be determined 
automatically by looking in the overmind database. 

With Overmind 

To use event detection in an overmind range, for each test script you wish to run with event detection, 
you’ll need to write your own test plan that calls the event detection harness with the arguments as 
explained previously. 

To make this easier, start with the following template plan: 

from tyworkflow. support . planlang import * 
test = TESTCASE( 

script = 'magnum. harness. evdet_harness', 
hostslots = [PUT_H0STS_HERE] , 
samples = -1, 
namespace='TEST_NAME-$t ' , 
paramslots = [ 

[ ' test=LEAFNODE_SPEC' ] , 

[ ' esxi_user=USERNAME ' ] , 

[ ' esxi_pass=PASSWORD ' ] , 

[ ' keep=@True' ] 

] 

) 

EXECUTE( 

testcase = test, 

) 

Save this code to a new tile with your other plans and scripts (DO NOT simply modify this tile in place), 
and modify the copy as follows: 

• In the hostslots parameter, replace PUT_H0STS_HERE with HOST objects according to the 
number and type of hosts your test script requires. (If your test script takes multiple hosts and 
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the host on which you want event detection is not the first host, remember to set the 
host_index keyword argument in paramslots). 

• If you want to have a default limit on the number of samples the plan will generate, then change 
the samples parameter. You probably just want to leave it at - 1, though. 

• In the namespace parameter, replace TEST_NAME with a very concise name of your test; this 
will be used as part of the default namespace name when submitting this plan. 

• In paramslots, 

0 Replace LEAFNODE_SPEC with the specification of your leafnode (using “python 
import path”, as before). 

0 Replace USERNAME and PASSWORD with the username and password used to log in to 
the ESXi hosts on your range. 

0 If you have a central vCenter server managing all the ESXi hosts in the range, you can 
define add the esxi_host parameter in paramslots with the vCenter server’s DNS- 
resolvable hostname or IP address as the value. Otherwise, leave esxi_host out and 
it will be determined by looking in the overmind database. 

o Add definitions for test_args (a list) and test_kwargs (a diet) to paramslots to 
define the positional and keyword arguments to be passed to your test script being run 
under event detection. 

At this point, you should now have a working test plan that will run your test script under event 
detection. You can run this test plan with remote commit like any other, and see its results in overview, 
except now, you will get screenshots as well. 
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